Improve comments and change PrimitiveArray::from_trusted_len_iter to take an ExactSizeIterator
#8564
Conversation
|
🤖 |
alamb
changed the title
PrimitiveArray::from_trusted_len_iter to take an ExactSizeIterator
| where | ||
| P: std::borrow::Borrow<Option<<T as ArrowPrimitiveType>::Native>>, | ||
| I: IntoIterator<Item = P>, | ||
| I: ExactSizeIterator<Item = P>, |
There was a problem hiding this comment.
what is the rationale for this change? I am thinking that maybe someone who had an iterator that knew its length but did not implement ExactSizeIterator would have to change their code
Maybe that is ok, but the implementation doesn't even seem to use any of the methods on ExactSizeIterator
There was a problem hiding this comment.
Yeah we just mentioned that as a side note in the last PR. I think it has two advantages:
- Users have at least some help from the type system to avoid safety issues.
- It's consistent with BooleanArray
However, I am also happy with the old API.
There was a problem hiding this comment.
I also think that some of the functions could make use of len instead of size_hint. If we choose to keep ExactSizeIterator I'll go through them.
I think another cause of incompatibility is caused by users needing to call .into_iter() manually.
There was a problem hiding this comment.
@mbrobbel Do you have any more thoughts on which iterator bound we should use?
There was a problem hiding this comment.
We should use TrustedLen, but we can't because it's nightly-only. So the next best things is ExactSizeIterator for iterators that know their exact length. However there is a note in the docs:
Note that this trait is a safe trait and as such does not and cannot guarantee that the returned length is correct. This means that unsafe code must not rely on the correctness of Iterator::size_hint. The unstable and unsafe TrustedLen trait gives this additional guarantee.
Since this method is marked unsafe we require users to make sure their iterators are TrustedLen. This means:
The iterator reports a size hint where it is either exact (lower bound is equal to upper bound), or the upper bound is None. The upper bound must only be None if the actual iterator length is larger than usize::MAX. In that case, the lower bound must be usize::MAX, resulting in an Iterator::size_hint() of (usize::MAX, None).
The motivation to use ExactSizeIterator here (I think) is to use ExactSizeIterator::len instead of Iterator::size_hint. This has the same "can't be None" requirement on the upper bound reported by the size hint.
For the bound we can use I: IntoIterator<Item = P, IntoIter: ExactSizeIterator>.
There was a problem hiding this comment.
|
🤖: Benchmark completed Details
|
|
Thanks for merging-in the latest main @alamb ! Do you see any reason why the integration tests are failing with these changes or was that a hiccup? |
probably the intermittent failure |
|
I merged up to resolve some conflicts |
|
run benchmark array_iter |
|
🤖 |
|
(whoops wrong PR) |
|
🤖: Benchmark completed Details
|
|
I already forget about this one! Is there any consent on whether we want to use If we don't want to change the API I can also close the PR to de-clutter the list. |
I would say in general if we don't see performance improvements we should probably close this PR / not make the change |
4 participants
Which issue does this PR close?
This is a follow-up to #8543 . There we discussed two outstanding issues that this PR tries to address.
Rationale for this change
Address the points from the PR.
What changes are included in this PR?
Vec-specific optimizations)ExactSizeIteratorfor thePrimitiveArrayAre these changes tested?
Existing test for
PrimitiveArrayAre there any user-facing changes?
Yes,
PrimitiveArray::from_trusted_len_iteris more restrictive.